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ABSTRACT 

Empirical norms are a traditional and accepted tool 
for reporting performance, other methods (e.g., statistical 
regression) are used more often in a researc" context to make 
conditional statements about performance gi^en a certain demographic 
context. This paper illustrates a method for the construction of 
differentiated school norms and the computation of percentiles on the 
basis of dempgraphic background factors. These differentiated norms 
facilitate reports of performance that take into account demographic 
conditions that are beyond the control of schools. With this approach 
each school is located at the median of its own norm group with 
regard to an index of socioeconomic status. The computation of 
percentiles is illustrated with science achievement test scores for 
all eighth-grade students in 1,573 public schools in California. The 
effects of varying the size of the norm group are illustrated with 
reading achievement test scores for all 12th-grade students in 797 
public high schools in California. The computing and reporting of 
differentiated norms are complex and may be difficult to communicate 
to general audiences. Local authorities in California, however, have 
reported few difficulties so far in communicating the concept of 
differentiated norms. Two tables and one figure illustrate the study 
data. (Author/SLD) 
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Abstract 



Empirical norms are a traditional and accepted tool for reporting 
performance. Other methods, e.g., statistical regression, are used more 
often in a research context to make conditional statements about 
performance given a certain demographic context. This paper illustrates s 
method for the construction of differentiated school normt and the 
computation of percentiles on the basis of demographic background factors. 
These differentiated norms facilitate reports of performance which take 
into account demographic conditions which are beyond the control of 
schools. With this approach each school is located at the median of its 
own norm group with regard to an SES index. The computation of percentiles 
is illustrated with science achievement test scores for a sample of 1,57** 
schools. The effects of varying the size of the norm group are illustrated 
with reading achievement test scores from a sample of 797 high schools. 
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A Method for the Construction 
of Differentiated School Norms 



Norms permit an assessment of the performance of an individual or 
group of individuals relative to a defined population. The norms can 
represent either an entire population or some meaningful subgroup. 
Commonly, norms are intended to represent an entire population although 
some test publishers offer norms based on a subgroup, for example, local 
norms which are based on individuals or groups in a limited geographical 
area. According to Angoff ( 1 984 , p. k$) , "the value of school mean norms 
can be enhanced if they are further differentiated in terms of school and 
community variables. 11 Cronbach (1984, p. Ill) notes that populations can 
be subdivided to allow differentiated comparisons of schools with reference 
groups of similar characteristics. An advantage of differentiated norms is 
that they can provide users with additional information which is helpful in 
understanding school performance. The objective of this study is to 
illustrate a method for constructing norms which take into account social 
and demographic characteristics «f schools. Two procedures which have been 
traditionally used for making differentiated comparisons of schools are 
briefly sketched below, including expectancy tables and linear regression. 
This is followed by a discussion and illustration of differentiated norms 
using science achievement scores and a comparison with the use of science 
residual scores. 
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One simple method of subdividing a population of schools is to compute 
and use a socioecononi c index to sort schools into fixed subgroups or SES 
ranges. Groups are defined on the basis of their location in fixed ranges 
of the value of a predictor, here a measure of socioeconomic standing. For 
each of these subgroups the distribution of the criterion variable is used 
to make normative judgements about school performance, i.e., the 
computation or a percentile rank. This is the expectancy tables described 
by Angoff (198*0. A disadvantage of expectancy tables is illustrated by 
the example of two schools that differ just enough on the SES index to be 
classified in different subgroups. The criterion performance of these two 
schools could be identical, yet their percentile ranks would differ, 
depending on the subgroup. If the SES measure is strongly correlated with 
the criterion, a change in subgroup can result in a large change in the 
percentile rank. This type of boundary effect is an issue when comparing 
the performance of schools in different SES subgroups. 

Another approach for making differentiated comparisons employs 
regression techniques. yer, 1 966 ; Dyer, Lynn and Patton, 1969; Klitgaard 
and Hall, 1973; Marco, 197^; Fetler and Carlson, 1985) For example, school 
mean achievement is predicted from socioeconomic or demographic information 
describing the pool of test takers. The difference between the predicted 
and the actual achievement, a residual or adjusted score, can be 
interpreted as a comparison with other statistically similar schools, and 
as the school's own effect on performance. Residual scores have long been 
used by educational researchers for making statistically controlled 
comparions of individual or group performance. But residuals have not so 



Differentiated School Norms 

5 

far replaced percentiles for communicating to the general pubiic and .>ch' 1 
of f ici als. 

The expectancy table method can be elaborated to construct 
differentiated norms for individual schools, (California Department of 
Educrtion, 1 988) Based on each school's unique social and demographic 
characteristics, a norm group is defined which consists of all other 
schools within a specified range in the distribution of a socioeconomic 
index. Unlike the expectancy table method, the range floats, so that each 
school is at the median of the distribution of the socioeconomic index f or 
its group. This group is smalle' and less diverse in terms of SES than the 
original population of schools. Percentiles are computed with reference to 
the differentiated norm group. Such percentiles provide a means for making 
normative judgements about schools which take into account social and 
demographic background factors. Direct comparisons with other schools in 
the group can be made. The construction of differentiated norms is 
i 1 lustrated below. 

Data Source 

The California Assessment Program annually administers a Science 
achievement test to all eighth grade students in 1573 public schools as a 
part of mandated statewide testing. A Reading achievement test is annually 
administered to all twelfth grade students in 797 public high schools, as 
well. School average achievement is expressed as a scaled score computed 
using I tern Response Theory techniques (Bock, Mislevy and Woodson, 1 982) . 
Background information is obtained from a student survey administered 
during the assessment. Students indicate on a five point scale the 
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educational level attained by his or her most educated parent: (1) not a 
high school graduate; (2) high school graduate; (3) some college; {k) 
college graduate; (5) advanced degreee. The parent education index (PEi) 
is the average of all student responses. Student mobility (MOB) was 
estimated by the percent of students newly enrolled in the district during 
the last year. English language fluency was estimated by the percent of 
test takers classified by teachers as limited English proficient (LEP) 
according to state mandated criteria. Finally, a measure of poverty, the 
percent of families receiving Aid to Families with Dependent Children 
(AFOC) was obtained for each school. 

Analyses 

The analytical steps described below include: the computation of a 
composite SES index; the use of the composite SES index to construct 
differentiated norms and science achievement percentiles; and the 
computation of science achievement residuals to be compared with the 
percentiles. The composite SES index for a school is estimated as the 
predicted score from a regression which the parent education index, student 
mobility, percent of limited English speakers, and AFOC are predictors, and 
the average of eighth grade reading and mathematics achievement is the 
criterion. Linear regression is one practical means of determining weights 
for the SES component measures in order to produce a composite index. 
Given that the predicted score can be interpreted as that part of 
performance attributable to, or explained by the SES components, the 
prediction can function itself as a measure of composite SES. Fetler and 
Carlson (1985) have shown that regression weights obtained in this way are 
stable both across years and academic subjects. This particular method of 
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computing an SES composite was chosen in order to obtain an index which is 
strongly correlated with the achievement test scores to be normed. 



Schools are ranked on the basis of the composite SES index. The 
differentiated norm group for a particular school includes both the 10 
percent of schools with immediately higher composite SES indexes and the 10 
percent with immediately lower indexes. The differentiated norm group for 
a school in the highest 10 percent of the SES distribution consists of the 
top 20 percent of schools. An analogous rule is used for the schools in 
the lowest 10 percent of the SES distribution. The middle 80 percent of 
schools stand at the median of the SES distribution for their 
differentiated norm groups, but do not necessarily fall at the median for 
other measures. Once the differentiated norm group has been specified, a 
percentile is computed for a performance measure, e.g., achievement, using 
conventional procedures. Similar procedures were applied to the high 
school Reading achievement scores, except that total group sizes of 10, 20 
and 30 percent were used. 



Linear regression was used to estimate the science residuals needed 
for comparison with the percentiles. The school average parent education 
index, student mobility, percent limited English speakers and AFOC were the 
predictors, and science achievement was the criterion. The following model 
was obtained: Predicted Science Achievement - ~ 0.8(AFOC) + 

4l.6(PEI) + 22.2(M0B) - 0.4(LEP). All parameter estimates were significant 
(p < .05) > with R-square equal to 0.5* The science residual equaled the 
actual minus the predicted score. 

Results 

O 
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Descriptive statistics and correlations are shown in 1^ e 1 for the 
composite SES index, science achievement, residual and percentile. SES 
correlated more highly with the achievement test score than with the 
percentile. Boundary effects in the highest and lowest differentiated norm 
groups are responsible for part of the correlation between the percentile 
and the SES index. Because schools in these groups are not at the median 
of the SES distribution there is a positive correlation between the SES 
index and the percentile. The SES index within each differentiated group 
does vary, which could also account for part of this correlation. The 
correlation between SES and achievement is smaller in the extreme groups 
than in the population. For the 333 schools (20 percent) with top ranking 
composite SES indexes, r = .W, and for the lowest ranking 333 schools, r = 
• 34. This deference results from a restriction in the range of SES in the 
differentiated group and diminished variance compared to the population. 
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TABLE 1: CORRELATIONS OF SCHOOL SCIENCE ACHIEVEMENT AND SES 



SES 
INDEX 

MEAN 505-6 
SD 69.25 
N 1571* 

SES INDEX 1 .00 

ACHIEVEMENT 0.73 

RESIDUAL 0.00 

PERCENTILE 0.13 



ACHIEVE- RESID- PERCENT- 



ME NT 


UAL 


ILE 


263. 1 


0.00 


50.22 


l»2.52 


28.93 


28. i*9 


1573 


1573 


1573 


0.73 


0.00 


0.13 


1 .00 


0.68 


O.72 


0.68 


1 .00 


O.90 


0.72 


0.90 


1 .00 



Descriptive statistics and correlations are shown in Table 2 for ranks 
obtained with the 10, 20 and 30 percent group sizes. The magnitude of the 
correlations suggest that manipulation of the group size had little effect 
on the ranking of schools. Even with larger groups most schools are still 
located at the median on SES and apparently remain at the same relative 
position on achievement. The ranks obtained wil.i larger groups did 
correlate more highly with SES, an undesirable property, than those 
obtained with smaller groups. By definition, with larger group size there 
are more schools at the extremes that cannot be at the SES median. 
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TABLE 2: CORRELATIONS OF RANKS OBTAINED WITH DIFFERENT GROUP SIZES 





10 % 


20 % 


30 % 


SES 




GROUP 


GROUP 


GROUP 


RANK 


MEAN 


50.2 


50.1 


1*9.9 


399 


SD 


28.6 


28.3 


27.9 


230 


N 


797 


797 


797 


797 


10 % GROUP 


1 .00 


0.98 


0.9** 


0.06 


20 % GROUP 


0.98 


1 .00 


O.98 


0. \k 


30 % GROUP 




0.98 


1 .00 


Q.Ik 


SES RANK 


0.06 


O.li* 


0.2^ 


1 .00 



A distribution of percentiles for an undifferentiated norm group is by 
definition rectangular. The empirical distribution of the percentiles 
computed here is itself rectangular with a computed skewness of 0.0 and 
kurtosis of -1.2. Examination of the cumulative frequency distribution of 
percentiles revealed 26 percent with values at or below 25, 50 percent at 
or beiow 50, and 75 percent at or below 75* Although this result appeared 
likely, given the relative complexity of the differentiated norm group 
approach, an empirical confirmation is of interest. 



Percentiles are plotted against science residuals in Figure 1. The 
relationship between the science percentile and residual resembles an 
elongated "S", roughly linear in the midranges and curved in the extremes. 
The curving in the extremes is mainly attributable to the upper and lower 
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bounds inherent in percentiles by contrast with the theoretically unbounded 
residuals* Examination of the plot suggests that the two methods produce 
results that are fairly consistent. There are however a number of visual 
outliers throughout. This does not necessarily argue Tor vhe superiority 
of one method over the other. The existence of such outliers suggests that 
for some schools varying judgments could be reached using the two 
analytical methods. Where practical decisions are being made, \:he 
existence of such differences may be grounds for closer inspection of the 
raw data. 

Descriptive statistics and correlations for the three ty|>es of twelfth 
grade ranks are shown in Table 2. The magnitude of the correlations 
between these types of ranks suggests that changes in group size, at least 
for the range of sizes investigated here, do not affect the ranking of 
schools. The percentiles obtained with larger groups are more strongly 
correlated with SES than those obtained with smaller groups. 

Di scussion 

Residual analysi and differentiated norms can both be used to make 
differentiated comparisons among schools. The differences in these^two 
methods suggest that they best serve different ends and a decision to use 
either should depend on specific needs. The use of residual analysis 
implies a comparison with other statistically similar schools. Part of the 
usefulness of this approach lies in the variety of statistical procedures 
available for examining, manipulating, making probability statements about, 
and interpreting residuals. However, residual analysis does not typically 
include the computation of percentiles and the associated normative 
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judgements, (See, for example, Draper and Smith, 1966) The use of 
differentiated norm groups involves a comparison with an empirical 
distribution of scores. The resulting percentile permits normative 
statements about performance which take background factors into account. 
The following discussion describes selected technical issues related to the 
use of differentiated norms, including: the treatment of very nigh and low 
SES schools; the procedures for constructing differentiated norm groups; 
determination of optimal group size; and interpretive difficulties. 
Finally, a question about the appropriate use of differentiated norms is 
raised. 

The problem with very high and low SES schools is that unique 
differentiated groups for them cannot be constructed and these schools do 
not fall at the median of the SES distribution of their group. Because SES 
correlates positively with performance the high SES schools will on the 
average receive an "unfairly" high percentile and the low SES schools will 
receive an "unfairly" low percentile, compared to other schools in the 
middle of the SES distribution. This problem is eased by the restricted 
range of SES in these extreme groups and the associated smaller correlation 
between SES and performance. 

One issue in the construction of differentiated norms is the 
computation cf an appropriate composite SES index. Here linear regression 
was used to produce a composite score that reflected non-school influences 
on achievement. One advantage of using this type of predicted score 
regression is that the SES index will correlate with the criterion as 
highly as is possible under a least squares model. If the goal of using a 
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differentiated norm is to adjust for SES, higher correlations are 
desirable. Other methods of constructing a SES index can be imagined, 
e.g., a purely logical evaluation of importance, or perhaps cluster 
analytic approaches could be used. 

Another Issue relates to the determination of optimal group size. 
Smaller groups permit finer differentiation and more appropriate 
comparisons. Larger groups, on the other hand, should result in in more 
stable estimates of the percentiles, other factors remaining equal. 
Manipulating the group size here did not change the ranking of schools. 
One interpretation is that whatever the group size, most schools are still 
at the median of their groups in terms of SES and maintain their relative 
position on achievement. The higher correlation of the percentiles with 
SES for the larger groups is related to the problems at the extremes of the 
SES distribution. With larger group sizes, the numbers of schools at the 
extremes, which cannot be at the SES median, increase. The results suggest 
that the determination of an optimal group size requires a balancing of the 
need for stable estimates of percentiles with the need to have percentiles 
which correlate at a low level with SES. In practice, both stability and 
correlation with SES will depend on the empirical distributions and 
covariance of the available measures. This implies that some trial and 
error investigation of different group sizes should precede any application 
of differential norms. 

Finally, The computing and reporting of differentiated norms is 
complex and will almost always require computer technology. This may be a 
problem when communicating results to audiences who are accustomed to 
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relatively simple, undifferentiated norms tables and percentiles which do 
not take into account background factors. On the other hand, some 
commercial test publishers will produce local norms for their clients on 
request. Local norms permit geographically differentiated comparisons with 
the performance of individuals or groups in a specific region. It is 
possible that the concept of differentiation on the basis of socioeconomic 
variables will as easily communicated as is the concept of local norms. 
Some support for this hypothesis may be found in the use made of 
differentiated norms by the California Department of Education (19&3) • 
Local school officials have reported few difficulties in communicating the 
concept of differentiated norms as described in this paper. 

The use of differentiated norms could motivate statements that a 
particular school's performance is below expectation or above expectation 
compared to other similar schools. For example, a school ! s residual score 
could be positive or negative, or its percentile might be greater or lesst 
than fifty. Once the precise criteria for making these statements are 
described, they have explanatory value. However, these explanatory 
statements are sometimes taken further to imply the value judgements that 
performance above expectation is good, and performance below expectation is 
bad. Such value judgements may neglect to take into account what is 
desirable for scnools in absolute terms thereoy encouraging an unmerited 
complacency or undeserved censure. For example, a low SES school could 
receive a percentile greater than 50 or a positive residual score, and 
still not meet minimally acceptable standards for achievement. Although 
the school average achievement could be greater than that of a majority of 
similar schools, it could still be that the educational needs of its 
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students are not v/el 1 met. Such instances underscore the importance of 
distinguishing the appropriate use of cnf ferentiated norms to explain 
performance from the value judgements which might bs made in addition. 
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FIGURE I PLOT OF SCIENCE ACHIEVEMENT PERCENTILE RANKS AND RESIDUAL SCORES 
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